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Abstract: The aim of this paper is to demonstrate a novel assessment method developed 
to determine if the curriculum from two separate safety degree programs provided 
sufficient opportunity for students to obtain the knowledge required for professional 
practice in occupational safety. The method relies on the Board of Certified Safety 
Professionals (BCSP) examination blueprints. In the graduate program case study, over 
88% of the BCSP criteria were met through an explicit means and up to 64% through 
assignments or better. Aggregating criteria into respective subject areas showed that the 
curriculum covered anywhere from 58% to 100% of the items within each BCSP topic. In 
the undergraduate case study, over 96% of the BCSP criteria through an explicit means, 
and 82.8% of knowledge items were assessed in assignments, exams or better. 
Aggregating criteria into respective subject areas showed that the curriculum covered 
anywhere from 75% to 100% of the items within each BCSP topic. Once briefed on the 
results, all faculty/instructors agreed that the approach helped identify strengths and 
weaknesses in their current curriculum. Most importantly, presentation of results acted as 
a catalyst for curricular discussions amongst the faculty that resulted in improvement 
priorities and a better understanding of student learning potential in course assignments. 


Keywords: professional development, professional practice, degree programs, 
curriculum, curricular discussions, student learning potential 


Introduction 

Since 2010, the American Society of Safety Engineers (ASSE), in cooperation with 
the Board of Certified Safety Professionals (BCSP), has increased their promotion of 
ABET-ASAC accreditation of occupation safety, safety and health, and environmental 
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safety and health degree programs. Some advantages of increasing the number of 
accredited safety degree programs include, but are not limited to: more current curricula 
and curricula improved through data analysis, required faculty development, evidence for 
more labs and lab equipment, better educated graduates, and the graduate safety 
profession (GSP) credential granted to graduates. However, the ABET-ASAC 
accreditation requires a considerable financial investment by the host-university and time 
and resources from the degree program faculty. Unfortunately, due to the depressed U.S. 
economy and reduced state budgets, universities are being required to reduce costs, 
postpone hiring, and simply “get by” with what they currently have. Due to the higher 
unemployment, college graduates are having difficulty gaining employment in their chosen 
field and national news outlets are questioning the value of a 4-year baccalaureate or 
terminal master’s degree. 

Although the occupational safety and health field employment outlook is rather 
promising (NIOSH 2011), degree programs across the country to reflect on the quality, 
value, and efficiency of their degrees in the face of budget issues. Safety degree programs 
across the U.S. are taking efforts to maintain the quality of instruction while reducing costs 
included efforts to more efficiently utilize instructional resources (e.g., cross listing of 
courses, sharing electives between programs, restructuring curriculums to reduce 
dependence on outside instructors) and develop consistency in admissions standards and 
program prerequisites across graduate programs. Whether considering curriculum—and 
program—level changes or committing to the accreditation process, faculty members in 
safety degree programs are forced to ask themselves: Are we in a position to invest our 
limited time and the universities’ limited resources to pursue program accreditation? 

Background 

Faced with increasing pressure to demonstrate degree value and justify the 
investment into pursuing degree program accreditation, department faculty from two 
separate safety degree programs (at two separate times) decided that an objective, 
externally based set of criteria was needed to validate the curriculum and provide evidence 
to support or argue against proposed curricular changes. The first program was a one-to- 
two year terminal master degree in environmental safety and health, located at a university 
in northern Minnesota. The other program offered both undergraduate and graduate 
degrees in occupational safety, located at a university in southern Wisconsin. The one 
commonality in these two program case studies was a single faculty member who worked 
at conducted the evaluations at both sites but at different times (approximately two years 
apart). ABET accreditation seemed a logical first step, especially given its critical role in 
the Graduate Safety Professional certification developed by the Board of Certified Safety 
Professionals for students in BCSP-approved graduate safety programs. But, as noted in 
the next section, ABET criteria for safety programs are also quite vague (ABET, 2008) and 
provide little specific guidance on curriculum content. Instead, the faculty needed to find 
some other set of criteria if they were to conduct an in-depth curriculum assessment. 
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It should be noted that this assessment was not focused on an outcomes 
assessment. Although the curriculum assessment results would eventually be used to help 
assess how well its students were learning the subject matter being presented in the 
curriculum, the faculty was more interested in the fundamental questions of curriculum 
assessment mentioned above. After all, a key assumption underlying outcomes 
assessment is that a good outcome measure indicates effective learning that, in turn, 
positively correlates with the professional quality and competence of a program’s 
graduates. However, what if a student learns a topic well, but the topic is irrelevant to 
practice? Or, what if a topic relevant to practice is only mentioned in the curriculum—or 
worse, not presented at all? Without proper curriculum assessment, outcomes 
assessment may reliably measure student learning, but runs the risk of being an invalid 
tool for assessing their professional quality and competency. 


In terms of assessing student learning in a higher education setting, the general 
assessment cycle appears to follow these steps (e.g., Allen, 2004; Diamond 1998; Maki 
2004): 

1. Establish new or review existing learning goals for the program 

2. Provide opportunities to achieve these goals 

3. Assess if students are learning from these opportunities to meet the goals 

4. Use information from assessments to adjust program (and thus 
competency) 

5. Repeat 

As inferred from above, the researchers were interested primarily in step 2 of this 
sequence - yet the literature search yielded little specific information on how to select and 
assess these “opportunities.” Rather, student learner outcomes were emphasized heavily 
while curricular content assessment received far less attention. 


In this context, learning outcomes can be considered the foundation for driving 
programmatic changes, but at least for STEM-based programs, most outcomes are 
adapted directly from ABET criteria for accreditation and are accordingly vague (e.g., “an 
ability to communicate effectively”). Worth noting as well, although the authors would 
general use the terms 'outcomes' and 'objectives' interchangeably, ABET clearly defines 
the terms for assessment (ABET, 2008): 

Program educational objectives [emphasis added] are broad statements that 
describe the career and professional accomplishments that the program is 
preparing graduates to achieve. 

Program outcomes [emphasis added] are narrower statements that describe what 
students are expected to know and be able to do by the time of graduation. These 
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relate to the skills, knowledge, and behaviors that students acquire in their 
matriculation through the program, (p. 1) 

Once again, the “skills, knowledge, and behaviors” are not defined in any way. In 
fact, it is the authors’ understanding that most accreditation and certification boards leave 
it to program faculty to decide what skills, knowledge, and behaviors should be included 
in their program in order to meet their stated outcomes. The typical argument behind this 
is similar to the one expressed by the National Council of Examiners for Engineering and 
Surveying (NCEES) when discussing the use of their Fundamentals of Engineering 
licensure exam for outcomes assessment, “[t]he exam should not, for example, be used 
to determine the curricular content of any program. Its purpose is to test competency for 
licensure; it is not intended to force programs to be similar” (Barrett et al., 2010, p.2). 

Specific to this program, the Educational Standards Committee of the American 
Society of Safety Engineers (ASSE) had worked with ABET in the mid-2000s to specify 
specific program criteria required to be in place if a program wanted ABET accreditation. 
Subsequently, this committee published guidelines for the broad topics to be included in 
a safety curriculum, but with a caveat similar to that expressed by the NCEES (ASSE, 
2011 ): 


The committee did not want to provide a long list of required courses or topics 
areas that were common in previous safety curriculum criteria by the [Board of Certified 
Safety Professionals] and ABET. The committee believes strongly that programs should 
be provided flexibility [...]. 

So, how do programs decide what skills, knowledge, and behaviors are needed? 
Anecdotally, most don’t assess their curricula at that level of detail. But, in programs that 
conduct detailed evaluations, curriculum mapping appears to be the most common tool 
used to make this decision (e.g., Soulsby, 2006). This method requires identifying what 
students do in their courses and what the faculty expects them to learn (the skills, 
knowledge and behaviors) and then clarifying the relationship between the two, or 
“mapping the curriculum.” This process reveals if a student’s learning opportunities are 
linked or consistent with faculty expectations. Inconsistencies suggest places for 
curriculum improvement that bridge the gap between the two and, in turn, increase the 
likelihood of meeting program objectives. However, to identify the skills, knowledge, and 
behaviors needed by a student, common practice is to glean information from a program’s 
stakeholders (e.g., faculty, administration, alumni, employers, funding agencies, peer 
programs, and professional societies), which can easily suffer from the same issue 
described earlier: everybody seems to provide a different opinion about what (they 
believe) students need to know when they graduate. 

Under the advice of each case study’s department faculty, the researchers went 
back to the ASSE Educational Standards Committee caveat mentioned prior. Although 
the ABET criteria had not proven to be helpful, the “long list” attributed to the Board of 
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Certified Safety Professionals (BCSP) turned out to be a promising lead. The BCSP has 
a primary mission similar to the NCEES: assess the professional competency of safety 
professionals via the Associate Safety Professional (ASP) and Certified Safety 
Professional (CSP) exams. Notably, certification and licensure agencies utilize recognized 
methodologies (e.g., ISO/IEC 17024 - Conformity Assessment) to ensure that their 
examinations test people on the activities, knowledge, and skills required in their 
profession. The key step in this process involves a job analysis of current practitioners. 
However, unlike the NCEES and many other licensure and certification agencies, the 
BCSP was very transparent in its exam development process, publishing highly-detailed 
“blueprints” outlining the skills and knowledge expected of a safety professional and from 
which the ASP and CSP exams were developed. 

The exam blueprints were derived from a three-stage job analysis study of current 
safety professionals, targeting 1500 survey subjects, with respect to the skills and 
knowledge needed to perform the safety job in a professional, competent manner (BCSP, 
2008b, 2008c). BCSP then categorized the resulting 249 knowledge items as either 
“foundation” (relevant to the ASP exam) or “advanced” (relevant to the CSP exam) and 
listed the 249 knowledge (and an additional 298 skill items) under a hierarchy of domains 
(e.g., risk management) and tasks (e.g., design effective methods to reduce or eliminate 
risk). Relevant to this initiative, the BCSP also undertook a generalized curriculum 
mapping effort, linking the skills and knowledge items with 15 “subject matter” topics 
typically taught in a safety program (BCSP, 2008d) - but provided no guidance on how to 
adapt this generalized curriculum map to a specific program. However, in a separate 
publication, one of the individuals involved in the original job analysis study did provide 
some guidance. Brauer (2005) not only described the job analysis survey and, in turn, 
suggested several ideas for using its results to assess a safety curriculum. With these two 
sources of information in hand, the program faculty now had an objectively derived set of 
skills, knowledge, and behaviors, as well as some ideas on how to assess the curriculum. 
All in all, the research efforts of BCSP created a comprehensive list of subject matter 
categories, (foundation and advanced) knowledge items, and skill items to use as a 
reference for the development of certification exams. The same certification exams that 
deem whether a person is a “certified safety professional” (CSP) or not. And it is this basic 
delineation that provided the basis for evaluating degree program curriculum based on 
evidence of course coverage and/or assignments. 

Study Objectives 

The overall goal of this study was to create and test a novel curriculum assessment 
method (based on 2004 BCSP Blueprints). Two types of safety degree programs 
(graduate and undergraduate), at different universities in the Upper-Midwest, were used 
in this study. The specific study objectives are: 

1. What percent or count of knowledge items is covered in course lectures? 
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2. What percent or count of knowledge items is covered in course assignments? 

3. Which knowledge items are covered most frequently? Or not covered at all? 

4. Which topic areas need attention for faculty discussion of curricular changes? 

Methods 

Two Case Studies - Two Unique Safety Degree Programs in the Upper-Midwest 

The assessment tool derived from BCSP’s 2008 Blueprints was first applied to a 
terminal master’s degree program in 2009. The master’s program had three core-faculty 
and four part- time or adjunct instructors, and generated approximately 10 graduates per 
year, all of whom earned full-time employment in safety after their capstone internship 
project. Based on lessons learned from the master’s degree program assessment, an 
updated version of the assessment technique was applied to an undergraduate degree 
program in 2011 (improvements/changes noted below). The baccalaureate program had 
six core-faculty and six part-time or adjunct instructors, and generated approximately 40 
graduates per year, most of whom gained full-time employment in safety after their 
capstone internship. 

In the graduate program assessment, both recent graduates and faculty were 
involved in the selection of topics and knowledge items rated within their fourteen courses 
(ten required courses and four elective courses). In the undergraduate program 
assessment, only faculty were involved in the selection of topics/knowledge items rated 
within their 22 courses (11 required courses and 11 elective courses). In the 
undergraduate study, faculty were also asked these questions: 

1. Do you dedicate a lecture (or portion of a lecture) to a discussion on ETHICS? 

2. Do you assign at least one (individually graded) writing assignment (paper) of 
10 pages or more? 

3. Do you assign (on average) 20 or more pages of reading per week? 

4. Do you assign group or team projects (either graded as a team or individually)? 

5. Do you assign student presentations (either graded as a team or individually)? 

6. Do you assign technical problem solving assignments (i.e. math-based 
problems)? 


Preparation for course assessment interviews. In both case studies, the 
capstone internship was not included in the assessment, because each student’s 
experience in that course was different. In addition, due to the sheer number of skill items 
and the difficulties in teaching and assessing skills in a traditional academic setting (most 
skill development occurs during actual practice, such as in an internship or after 
graduation, although lab experiences mitigate this to some degree), the faculty decided to 
exclude the 298 skill items identified by the BCSP from the assessment and focus 
exclusively on the 249 knowledge items. 
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Faculty recognized that evaluating each course against the full 249-item list would 
take too much time and effort, contributing to half-hearted responses or unwillingness to 
participate throughout the entire project. Therefore, the first step was to narrow down the 
list by deciding which knowledge items from the BCSP blueprints were relevant to each 
course, so only those relevant items would be rated for coverage and/or application in an 
assignment. The faculty also recognized, however, that the course instructors and 
students may differ in whether or not they perceive a knowledge item to be relevant for a 
course. In this context, a knowledge item’s perceived relevance to a course could fall into 
one of three categories: “definite” (all respondents reported the item as relevant to the 
course), “likely” (at least half, but not all, of the respondents reported the item as relevant), 
and “possible” (less than half, but at least one, of the respondents reported the item as 
relevant). This suggests that, if all respondents report an item as relevant (“definitely”), a 
reasonable course expectation is that the item would be covered in some depth; but, if 
only one respondent lists the item as relevant (“possible”), minimal coverage of the item 
would be sufficient or even unnecessary. Accordingly, the perceived relevance of each 
knowledge item played a key role in the aggregate data analysis and in selecting 
recommendations for improving course coverage of a specific subject area or knowledge 
item. 

Creation of Data Collection Instruments 

The resulting methodology required two separate rounds of data collection, each 
with its own data collection instruments: one for selecting topic areas or knowledge items 
to be rated within each course and another for the customized interview with instructors 
to rate knowledge items based on evidence of coverage and/or student learning (i.e. 
exams, projects, papers, etc.). 

The first round of data collection consisted of a cover letter describing the study 
and instructions for users to select the knowledge items (sorted by subject area) relevant 
to the specified courses based on the original presentation of the BCSP blueprints. The 
cover letter and Excel spreadsheet (containing the course description in one tab and all 
the BCSP knowledge items in a second tab) were sent to a group of students who recently 
completed the courses, the course instructor, and were also completed by the student 
research assistant (whom also recently completed all the courses). 

The second round of data collection consisted of an interview with each course 
instructor using a customized datasheet based on the results of the first round of data 
collection. The customized interview datasheet had three sections: knowledge items with 
“definite” relevance to a course, knowledge items “likely” to be relevant to the course, and 
knowledge items “possibly” relevant to a course. Within each section, items were grouped 
by subject area and each item was preceded by a text box in which the item’s coverage 
rating could be entered. 
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Data collection. The first round of data collection was driven by personalized e- 
mail messages, with the cover letter language in the body of the e-mail and a self- 
administered survey instrument as an Excel attachment. After the initial submission, 
several reminder e-mails were sent to all respondents encouraging participation (this was 
primarily for the student volunteers as all faculty had already agreed to participate and did 
so immediately). After a week, the student research assistant tallied the responses, and 
created customized interview spreadsheets for each course with the relevant knowledge 
items, grouped by subject area. 

In the second round of data collection, the student research assistant helped the 
instructors rate the course coverage of each knowledge item listed for the course, 
regardless of relevance level. This entailed face-to-face interviews, conducted individually 
with each course instructor who had been instructed to be inquisitive and acquire 
anecdotal evidence supporting each knowledge item rating. Based on course instructor 
evidence of coverage and/or assignment (exam, project, paper, etc.), each knowledge 
item was rated on a scale of 0 through 5 (in the graduate case) or 0 through 7 (in the 
undergraduate case). The ratings were entered into a single master spreadsheet, with 
courses across the column headings by item priority area (four columns per course, three 
priority and “no rating”) and the 249 knowledge items horizontally (by subject area) by row. 


Table 1. Rating Scale Use for Graduate Case 


Evidence of Item Coverage Within a Course Rating Scale: (0-5) 

5 

Thoroughly covered in lecture. Projects, presentations, quizzes, tests or other tangible 
products were utilized to assess knowledge item. 

4 

Discussed extensively in lecture. Material related to the item was included in homework 
assignments or quizzes to assess the level of knowledge acquired. 

3 

Item was covered in the course and included in notes, slides, handouts, activities, etc. 
However, students were neither tested nor asked to demonstrate their understanding of the 
item. 

2 

The knowledge item may not have been covered or discussed in lecture, but was included in 
assigned reading material. 

1 

Although possibly relevant, the item was not covered in any way in the course. 

0 

The knowledge item is not relevant to this course. 


Table 2. Rating Scale Used for Undergraduate Case 


Evidence of Item Coverage Within a Course Rating Scale: (0-7) 

7 

Individual research paper or applied project 

6 

Group research paper or applied project 

5 

In-class exam, quiz, or assignment 

4 

Homework, take-home quiz/exam 

3 

Lecture topic or assigned reading 

2 

Identified in syllabus or book, not necessarily covered or stressed in lecture 

1 

Not covered in class, no evidence of coverage 

0 

Not scored or Not applicable 
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Data analysis. After considerable trial and error, the following data analysis 
approach was selected as optimal for extracting reliable results from the master 
spreadsheet. The first step was to calculate the total number of course ratings given to 
each knowledge item, and then sorting the ratings by their perceived relevance (rating). 
Within each level of relevance, the ratings equal to 3 or higher (per knowledge item) were 
tallied. From the resulting table of rating frequencies, the total percent of knowledge items 
by coverage rating and relevance level were calculated to provide an estimate for the 
quality of knowledge item coverage across the program curriculum. The percentages for 
each subject matter area were then calculated (percent coverage and ratings of 
knowledge items within subject area). 


The second step consisted of transferring the data from each subject area 
worksheet into a new table to evaluate the aggregated ratings for the associated 
knowledge items and identify subject area coverage by course. The calculation of total 
counts and percentages was similar to the methods used on the entire data set. For each 
subject area, a bar chart illustrated the percent of knowledge items covered adequately 
(rating of 3 or greater) and inadequately (rating of 2 or less). Because items not receiving 
any ratings (“not rated”) were excluded from this analysis, the sum of the adequate and 
inadequate percentages within a course fell between zero and one-hundred percent. This 
allowed the knowledge items with low ratings to be easily identified visually and noted for 
later consideration. In addition to the bar charts, two matrices were created containing 
each course’s percentage of adequate coverage and inadequate coverage for each 
knowledge item. Within each matrix, courses were listed on the horizontal (top) and the 
subject areas were listed on the vertical (left). The matrices were then populated with the 
respective coverage ratings and color-coded to indicate areas receiving either above or 
below a specified threshold. This helped identify which courses either covered a topic well 
or should be considered for material additions or structural changes to improve subject 
area coverage and/or type of assignment to potentially assess (in the future) student 
learning. The following sections describe in more detail how the resulting data was 
interpreted and pilot-tested for curricular improvements. 


Results 

Case Study 1: Graduate Program - Individual Item Coverage Within Courses 


Each individual course was rated based on items selected by faculty/alumni: 60 to 
124 knowledge items out of a total of 249 items (Figure 1). Results of the entire curriculum 
assessment indicated that an individual course might cover anywhere from 5 to 26 
“definitely” relevant topics, another 9 to 51 topics “likely” to be relevant to the course, and 
anywhere from 10 to 89 additional topics “possibly” relevant to the course. 
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Item coverage by course 



Possible 

■ Likely 

■ Definite 


Figurel. Item coverage by course and perceived relevance of the item to the course. 


Graduate Program - Individual Item Coverage Across the Curriculum 

Of the total 249 knowledge items, only three items (business management 
software, Poisson distributions, and agricultural/food supply safety) were not identified for 
course rating in any of the 14 courses evaluated. On the other hand, 11 items were 
covered at some level in at least 10 of the 14 classes (including education and training 
methods, several types of administrative hazard controls, facility safety principles, and 
hazard identification); no items appeared in all 14 courses. For example, 13 courses 
covered some aspect of “Administrative controls: Written plans, procedures, and work 
practices.” Results of coverage rating and analysis indicated this topic was considered 
definitely relevant to two courses, and those two courses covered the topic through a 
written assignment (coverage rating = 5), whereas the topic was considered likely relevant 
to six other courses and possibly relevant to five others. Of the six courses for which the 
topic was likely to be relevant, two were reported as having covered the topic with a paper 
assignment. Of the remaining five courses for which the item was possibly relevant, one 
course was reported as having covered the topic to the highest rating—but, exemplifying 
the pattern reported the end of the prior section, the instructor reported this while the 
students did not consider the topic as relevant. 

Overall, further analysis (Table 3) suggests that the program delivered up to 88% 
of the BCSP knowledge items through an explicit means, such as lecture or other in-class 
activity, indicated by a quality of coverage rating of 3 or higher. Two-thirds of all items 
(64%) were thoroughly covered in the courses (rating = 5), indicating that these topics 
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were not only covered in class, but also through projects, tests, and other methods to help 
students demonstrate learning. 


Table 3. Graduate Case - Knowledge Item Coverage by Subject Matter Area 


Subject Matter Area 

Total 

number of 
items in 

area 

Number 
of items 
explicitly 
covered in 

a course 
(rating>2) 

%of 
items 
explicitly 
covered in 

a course 
(rating>2) 

Number 
of items 
covered 
with 
Assign. 
(rating>3) 

%of 
items 
covered 
with Assig 
(rating>3) 

Business Mgmt Principles 

28 

20 

71.4% 

11 

39.3% 

Ergonomics, Human Factors Science 

11 

11 

100% 

9 

81.8% 

Emergency Mgmt 

8 

7 

87.5% 

6 

75.0% 

Environmental Sciences 

18 

17 

94.4% 

12 

66.7% 

Education, Training, Communication 

23 

22 

95.7% 

18 

78.3% 

Fire Sciences 

11 

10 

90.9% 

10 

90.9% 

General Sciences 

8 

5 

62.5% 

5 

62.5% 

Hazard Recognition and Control 

44 

44 

100% 

41 

93.2% 

Health Sciences 

18 

17 

94.4% 

11 

61.1% 

Industry-specific Safety Principles 

12 

11 

91.7% 

8 

66.7% 

Measurement/Monitoring 

5 

5 

100% 

5 

100% 

Organizational/Behavioral Sciences 

10 

10 

100% 

5 

50.0% 

Risk Assessment and Risk Mgmt. 

18 

18 

100% 

10 

55.6% 

EHS Mgmt and Auditing Systems 

12 

7 

58.3% 

3 

25.0% 

Security Sciences 

23 

18 

78.3% 

6 

26.1% 

Total items 

249 

222 


Total coverage across all knowledge items 

89.2% 

64.3% 

Average coverage within subject matter areas 

88.3% 


64.8% 


Graduate Program - Items That Need Faculty Attention 

Aggregating the results into the respective subject matter areas showed that, 
within the each of the 15 subject matter areas, anywhere from 58% to 100% of the 
relevant knowledge items were explicitly covered in a course (i.e., quality of coverage 
rating equal to 3 or better). For example, the curriculum explicitly covered all the 
knowledge items in four subject areas (ergonomics, measurement/monitoring, 
organizational/behavioral sciences, and risk assessment/ management) but failed to 
adequately cover between 20-42% of the items in another four subject areas (business 
management principles, general sciences, EHS management and auditing systems, and 
security sciences; Table 4). Each of these latter subject areas was investigated further by 


42 


T. W. Loushine and R. G. Feyen 












www.hlrcjournal.com 



Access 


identifying coverage (or lack of it) within individual courses for each knowledge item within 
a subject area. At this point, the analysis could also have explored knowledge items with 
coverage limited to lecture or assigned reading (rated 3). Although these items are 
covered, no evidence of student learning exists for them, so it is in the best interest of 
faculty to discuss these items (Table 5). 


Table 4. Graduate Case - Knowledge Items Not Covered Adequately 


Topic Area 

Knowledge Items not covered adequately, rating<3 (n=26) 

BMP 

Budgeting, finance, and economic analysis techniques 

BMP 

Business planning and business continuity and contingency planning 

BMP 

Business software 

BMP 

Definition and use of net present value 

BMP 

Financial management principles 

BMP 

Project management software 

BMP 

Project management terminology 

BMP 

Schedule management principles 

EM 

Dispersion modeling 

ES 

Dispersion modeling 

ETC 

ANSI/ASSE Z490.1 

FS 

Structural and mechanical hazards 

GS 

Concepts of probability 

GS 

Poisson distribution: description, calculations, and interpretations 

HS 

Agriculture safety-including food supply safety 

ISP 

Product safety 

SHES 

Purpose and objective of ANSI/AIHA Z10 

SHES 

Purpose and objective of ISO 19011 

SHES 

Purpose and objective of the ISO 14000 series of environmental management system standards 

SHES 

Purpose and objective of the OHSAS 18000 series of OHS management system standards 

SHES 

Purpose and objective of the U.S. OSHA Voluntary Protection Program (VPP) 

SS 

Administrative controls 

SS 

Biological hazards 

SS 

EC: Engineering controls (general) 

SS 

Natural hazards 

SS 

Personal protective equipment 

Non-bold = Foundation Knowledge items 

Bold = Advanced Knowledge items 

(as per evaluating safety-related academic curricula using the content and role delineation analyses conducted for 
professional safety practice, BCSP Technical Report 2008-3, Jan. 2008) 
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Table 5. Graduate Case - Knowledge Items Only Covered in Lecture 

Topic Area 

Knowledge Items Rated Only "3" (n=28) 

BMP 

Definition and use of return on investment 

BMP 

Information security and confidentiality requirements 

BMP 

Lagging indicators 

BMP 

Leading indicators 

BMP 

Management sciences 

EHFS 

Competencies of other professionals with whom the safety professional interacts 

EM 

Competencies of other professionals with whom the safety professional interacts 

ES 

Competencies of other professionals with whom the safety professional interacts 

ETC 

Product certification and listing agencies 

ETC 

Sources of information on risk 

ETC 

Sources of information on risk management options 

GS 

Normal (Gaussian) distribution: description, calculations, and interpretations 

HRC 

EC: Segregation and separation 

HRC 

Pressure relief systems 

HRC 

Structural and mechanical hazards 

HS 

Administrative controls 

HS 

Epidemiology 

ISP 

Mining safety 

OBS 

Methods of achieving project stakeholder acceptance of project goals 

OBS 

Stakeholder participation committees 

RARM 

Chain of custody procedures 

SS 

Chain of custody procedures 

SS 

Chemical hazards 

SS 

Competencies of other professionals with whom the safety professional interacts 

SS 

Information security and confidentiality requirements 

SS 

Methods and techniques for evaluating facilities, products, systems, processes, and 
equipment 

SS 

Structural and mechanical hazards 

SS 

Transportation safety and security 


Non-bold = Foundation Knowledge items 
Bold = Advanced Knowledge items 

(as per evaluating safety-related academic curricula using the content and role delineation analyses conducted 
for professional safety practice, BCSP Technical Report 2008-3, Jan. 2008) 
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Case Study 2: Undergraduate Program - Individual Item Coverage within Courses 


Each individual course was rated based on knowledge items within subject areas 
selected by the most recent instructor for that course: 3 to 15 subject areas or 11 to 186 
knowledge items out of a total of 244 items (Figure 2). The lower number of knowledge 
items per course was the results of choosing subject areas and not individual knowledge 
items for rating a course. This made the pre-selection process quicker and easier, and 
reduced some confusion during the interview and in analysis. Another difference from 
case study 1 is that the rating scale went from 0 to 7 (instead of 0-5). Although the scale 
is not a truly validated scale, a rating of 6 or 7 would be considered consistently better 
than a 4 or 5 because when a student produces a paper or written project, the instructor 
can assess several forms of student learning. The fluctuation in number of items rated 
per course was the result of course instructor perception and the unique nature of certain 
elective courses (such as Safety 472 - Advanced Ergonomics). 



Figure 2. Item coverage by course and perceived relevance of the item to the course. 


Undergraduate Program - Individual item coverage across the curriculum. 

Another improvement over the graduate case study was that all items were rated several 
times due to instructors choosing topic areas instead of individual knowledge items. In- 
depth analysis (Table 6) suggests that the program delivered up to 96.3% of the BCSP 
knowledge items through an explicit means, such as lecture or other in-class activity, 
indicated by a quality of coverage rating of 3 or higher. Over four-fifths of all items 
(84.8%) were thoroughly covered in the courses (rating = 4 to 7), indicating that these 
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topics were not only covered in class, but also through projects, tests, and other methods 
to help students demonstrate learning. 

Table 6. Undergraduate Case - Knowledge Item Coverage by Subject Matter Area 


Subject Matter Area 

Total 
number 
of items 
in area 

Number of 

items % of items 

explicitly explicitly 

covered in a covered in a 

course course 

(rating>2) (rating>2) 

Number of 
items 
covered 
with Assig. 
(rating>3) 

% of items 
covered with 
Assign. 
(rating>3) 

Business Mgmt Principles 

28 

24 

85.7% 

20 

71.4% 

Ergonomics, Human Factors Science 

11 

11 

100% 

11 

100% 

Emergency Mgmt 

8 

7 

87.5% 

7 

87.5% 

Environmental Sciences 

18 

18 

100% 

16 

88.9% 

Education, Training, Communication 

23 

22 

95.7% 

18 

78.3% 

Fire Sciences 

10 

10 

100% 

9 

90.0% 

General Sciences 

8 

6 

75.0% 

4 

50.0% 

Hazard Recognition and Control 

41 

41 

100% 

41 

100% 

Health Sciences 

18 

18 

100% 

18 

100% 

Industry-specific Safety Principles 

12 

12 

100% 

11 

91.7% 

Measurement/Monitoring 

4 

4 

100% 

3 

75.0% 

Organizational/Behavioral Sciences 

10 

10 

100% 

6 

60.0% 

Risk Assessment and Risk Mgmt. 

18 

18 

100% 

15 

83.3% 

EHS Mgmt. and Auditing Systems 

12 

12 

100% 

11 

91.7% 

Security Sciences 

23 

22 

95.7% 

17 

73.9% 

Total items 

244* 

235 


207 


Total coverage across all knowledge items 

96.3% 


84.8% 


Average coverage within subject matter areas 

96.0% 


82.8% 


*lt was discovered during interviews that some items appeared to be very similar, so they were 
combined. 


Undergraduate Program - Items that need faculty attention. Aggregating the 
results into the respective subject matter areas showed that, within the each of the 15 
subject matter areas, anywhere from 75% to 100% of the relevant knowledge items were 
explicitly covered in a course (i.e., quality of coverage rating equal to 3 or better). The 
results were sorted down to individual knowledge item ratings to identify which items were 
rated below 3 (not covered in a lecture or reading; Table 7) or those that received no 
higher rating than 3 (only covered in lecture; Table 8). The topic areas under 100% and 
the individually identified knowledge items were identified and prioritized for discussion 
with faculty about what could be done to improve coverage and/or develop course 
assignments to evaluate student learning. 
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Table 7. Undergraduate Case - Knowledge Items Not Covered Adequately 


Topic Area 

Knowledge Items not covered adequately (n=9) 

BMP 

BMP 

BMP 

BMP 

ETC 

EM 

GS 

GS 

SS 

Financial management principles 

Information security and confidentiality requirements 

Project management software 

Project management terminology 

ANSI/ASSE Z490.1 

Dispersion modeling 

Inferential statistics: description, calculations, and interpretations 

Poisson distribution: description, calculations, and interpretations 

Radiation hazards 


Table 8. Undergraduate Case - Knowledge Items Only Covered In Lecture 


Topic Area 

Knowledge Items Rated Only "3" (n=28) 

BMP 

BMP 

BMP 

BMP 

ETC 

ETC 

ETC 

ETC 

ESHMS 

ENV 

ENV 

FIRE 

GS 

GS 

IS 

MM 

OBS 

OBS 

OBS 

OBS 

RARM 

RARM 

RARM 

SS 

SS 

SS 

SS 

SS 

Chanae management 

Definition and use of life cycle cost 

Definition and use of net present value 

Schedule management principles 

Competencies of other professionals with whom the safety professional interacts 
Methods of training delivery 

Methods of training evaluation 

Training assessment instruments 

Standards development processes 

Personal protective equipment 

Qualitative, quantitative, deductive, and inductive risk assessment methods 
Competencies of other professionals with whom the safety professional interacts 
Concepts of probability 

Descriptive statistics: description, calculations, and interpretations 

Competencies of other professionals with whom the safety professional interacts 
Electronic data transfer methods and data storage options 

Group dynamics 

Methods of facilitating teams 

Multidisciplinary teamwork 

Negotiation procedures 

Chain of custody procedures 

Insurance/risk transfer principles 

Sources of information on risk management options 

EC: Engineering controls (general) 

Measurement and monitoring 

Methods & techniques for evaluating facilities, products, systems, processes, & equip. 
Personal protective equipment 

Structural and mechanical hazards 
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Discussion 

As the results suggest, both degree programs demonstrated that their curriculum 
covered most of the BCSP knowledge items, with fairly well-defined areas for 
improvement. The analysis revealed and prioritized subject areas and knowledge items 
for faculty discussion about improving evidence of student learning. The discussion about 
individual course content and how student learning is assessed through assignments 
actually began in the interview process, because instructors had to provide evidence and 
explanation to support their rating of knowledge items. The overall results show “overlap” 
and “gaps” in the overall degree program that can be further analyzed and discussed by 
faculty to determine the best means to improve curriculum to conserve resources and 
time. 


In the case of the graduate program, the internal auditing procedure revealed a 
possible tool for outcomes assessment, even though that was not the focus of the 
assessment. Consider the occasional disconnect reported between the instructor and the 
student respondents in terms of knowledge item relevance. Two possible reasons may 
explain this: one, perhaps an instructor may have planned to present on an item, yet for 
whatever reason, did not do so during the most recent semester. Or perhaps, students 
didn’t learn the item sufficiently to understand its relevance to the course. Either way, 
instructors could assess the effectiveness of knowledge item coverage by comparing their 
perceptions of item relevance with student perceptions. Instructors could also use the 
information over multiple semesters to explore how the use of different topics or modes of 
presentation might fill gaps between their intentions and the students’ actual exposure to 
the material. 

With regards to the curriculum assessment and the information gleaned from the 
results, the program faculty felt strongly that this approach using professional certification 
criteria is promising, but could still use some fine-tuning. The biggest disadvantage is the 
time and effort required. On average, the surveys and interviews took about one to two 
hours per course, and each participant had to take their task seriously to provide accurate 
information. 

One other major difficulty encountered was that the terminology used in the BCSP 
documentation (e.g., blueprints, curriculum mapping, etc.) was sometimes unclear and 
redundant. BCSP personnel were helpful, but often were not able to provide sufficient 
clarification on what was meant by a specific skill or knowledge item. Nonetheless, the 
consensus definitions may represent a concept differing from the knowledge item 
originally captured in the original BCSP job analysis study. 

Despite these challenges, the approach clearly helped the program answer the 
questions raised earlier in terms of whether or not program graduates are exposed to the 
material they should know in order to practice as EHS professionals. This curriculum 
assessment methodology provided answers at several levels by providing baseline 
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measurements of knowledge item coverage both within individual courses and in the 
overall program. Within courses, as hinted earlier, the level of item coverage—particularly 
for items identified as definitely or likely to be relevant to the course—can be easily 
examined and modified if necessary. 


At the program level, the approach allows faculty to more effectively explore the 
impact of both temporary and permanent changes to the program curriculum—particularly 
in light of recent economic constraints. Using this approach provides program faculty with 
an opportunity to objectively evaluate the impact of adding or dropping classes from the 
curriculum, changing the frequency of course offerings, and whether a class should be 
required or can be offered as an elective. Further, in terms of program objectives, the 
faculty was able to seriously consider examining knowledge item coverage in terms of 
“foundation” and ‘advanced” items to distinguish how well the program prepares graduates 
not only for professional practice, but either the ASP or the CSP certification exam. 


Perhaps more importantly, the BCSP foundation provides a significant degree of 
objectivity to curriculum assessment. Rather than rely on feedback from numerous 
stakeholders in the program, each with different agendas and conflicting opinions, the 
knowledge items used in this approach are derived from a profession-wide job analysis 
study conducted in compliance with an accepted international standard (ISO/IEC 17024) 
and utilizing data collected in three stages, that targeted 1500 survey subjects from 
practicing EHS professionals (BCSP, 2008a). Regardless of academic institution, the vast 
majority of faculty would not be unable to perform a study of this depth for their program. 
This externally validated set of skills and knowledge items removes the discussion about 
the value or necessity of certain topics (e.g., between college administration and program 
faculty) from the realm of opinion. 


Because BCSP has been open with publishing the skills and knowledge items sets 
derived from their job analysis studies, this approach can be readily adapted to any EHS 
program. In a move that should simplify this assessment methodology and help address 
the concern regarding interpretation of terminology within the knowledge items, the BCSP 
has since simplified its blueprints, reducing the number of domains and more clearly 
detailing the knowledge and skills areas within those domains (BCSP, 2009). Interestingly, 
the approach could also be adapted to other degree programs with a certification or 
licensing body such as in the engineering disciplines. For example, the NCEES regularly 
conducts PAK (professional activities and knowledge) studies for the various engineering 
disciplines as part of its exam development process (e.g., NCEES, 2011). However, 
NCEES does not make these studies readily available and, should an engineering 
program wish to utilize this approach, they would have to obtain access from NCEES to 
the latest PAK data for their discipline. 
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Conclusions 

Overall, reflection on this curriculum assessment process identified opportunities 
for improvement, including streamlining the data collection and analysis process and 
clarifying the meaning of individual BSCP knowledge and skill items. But, ultimately, by 
using the certification agency’s job analysis data to indicate the knowledge needed by 
graduates of a safety program and developing a combined rankings and ratings 
methodology to assess coverage of this knowledge, the faculty was able to satisfactorily 
answer the initial question: Are we in a position to invest our limited time and the 
universities limited resources to pursue program accreditation? The authors believe that 
for both case studies, although changes need to be made and a decent amount of effort 
is required, that pursuit of ABET-ASAC accreditation be warranted. Other EHS- related 
programs could employ this new approach, and perhaps even other disciplines, to answer 
the same question and make fine-tuning improvements to the methods. 

Limitations 

The use of the BCSP blueprints assumed that subject areas and items 
encompassed the breadth of knowledge required of students graduating from an 
accredited degree program. This may not be the case, but an argument can be made that 
curricula that covers near 100% of material planned to be on a professional certification 
exam means that students of that degree program are at an advantage to passing the 
exam. And that those students, if they kept good lecture notes and/or created detailed 
course binders, would not have to invest in costly study materials or attend study review 
sessions. The rating scales used to assess how well a knowledge item was either covered 
in a course or potential to assess student learning was not validated. From a logical 
standpoint, the greater the time and intensity used to study or develop written projects, the 
more likely a student would both understand (and possibly apply) and remember that 
information. However, this is not true 100% of the time, and again, is only a theoretical 
assumption for these case studies. All in all, the faculty received an inexpensive review of 
their program curricula with recommendations on what they can do to improve topic 
coverage in courses and identify critical assignments in courses that demonstrate student 
learning outcomes, which are a key metric in ABET-ASAC accreditation reviews. 
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